Watch Stephan and Avinash’s webinar as a streaming Flash video »

Or, alternatively download/watch as a Quicktime (m4v) movie (77 MB).

Read the entry submission and GravityStream product overview published by the NZ Marketing Association and written by Netconcepts’ very own Chief Executive Officer, Nigel Varcoe.

Interview conducted by Brian Quinton, Senior Editor of DIRECT Magazine on Thursday, February 1, 2007.

The problem is you are staring at the tip of the iceberg, oblivious to the iceberg below. Something gigantic lies beneath. And you need more sophisticated tools and methods to understand and diagnose the opportunity in order to take appropriate action. In fact, chances are high that your natural search channel is seriously underperforming as a result.

As NSO best practices become commonplace amongst your competition, taking this evolutionary step is required for merchants to gain advantage. Fortunately, by approaching the natural search channel with an understanding of long tail dynamics, new key performance indicators (KPIs), and yield management techniques, savvy merchants can move beyond the traditional project-based mentality of NSO, to a more sophisticated, yet DM-friendly approach – one that leverages website size, brand power, and direct-marketing principles to capture the enormous potential that lies beneath the surface.

The Long Tail of Natural Search

To understand the KPIs and management techniques, let’s examine what lies beneath the surface. In a word, it is not an iceberg, but the “long tail” - a colloquial term referring to a common statistical distribution featuring a “tail-shaped” curve. Thanks to Chris Andersen’s book, the long tail concept has received much press recently, and carries many significant Internet business-based implications. But there is also a long tail distribution that contains the promise of natural search for large, dynamic e-commerce merchants.

Source Netconcepts, 2006

In fact, to appreciate the enormity of this potential, contrasting it against a known quantity is helpful. According to research published recently by Netconcepts (“Chasing the Long Tail of Natural Search”, 2006) for every search that occurs for the average merchant’s brand name, nearly 40 relevant searches occur for more generic, brand-neutral keywords. For instance, if there are 100,000 searches for “LL Bean” this month, there are an estimated 4,000,000 searches for hundreds of thousands of “unbranded” keyword markets LL Bean pages could compete within, ranging from “furniture slipcovers” to “women’s flannel pajamas” to “men’s reindeer sweater,” and more.

For these coveted and targeted keywords, LL Bean is likely to have matching category or product pages somewhere on the website, that ideally could position the LL Bean brand high enough in the search results to win these millions of click decisions (and sales) on the cheap. And even if these pages do not convert 100% of these searches into clicks, if well ranked, these pages still serve the purpose of building brand by association with these millions of corresponding “unbranded” searches, and by extension, the searchers that used them.

This is the long tail of natural search – the universe of diverse, unbranded keyword markets that, while perhaps not as frequently searched on an individual basis, cumulatively, add up to orders-of-magnitude greater (40x, in our research) search potential than the brand search traffic that most merchants naturally receive. This is the prize every savvy merchant seeks.

Since brand popularity is relative, a more objective way to think of this long tail potential is as a function of your website size. In particular, our research suggests that for every unique page that exists on a merchant’s website, nearly 100 unbranded searches are conducted in an average month. If you have a 20,000 page website, your long tail potential would be calculated as 2,000,000 unbranded searches, and so on. Tracking your unique pages and their yield therefore is a critical measurement to understanding the dynamics of your natural search channel performance.

Capturing the Elusive Tail

The long tail of unbranded keywords is a new entity for most marketers to grapple with. What drives it? How do you quantify it? What is good performance? What is bad? How do you capture it?

We set out to answer these questions in the research alluded to earlier. To this end, we developed a theory we refer to as “Page Yield Theory.” Page Yield Theory aims to breakdown, quantify, and model the components of a merchant’s long tail into manageable units, and was informed via natural search data gathered from a few dozen top online merchant (clients utilizing our GravityStream™ natural search optimization proxy technology). In the process, we developed a set of metrics to understand the dynamics of the long tail. We have since applied these same ratios as key performance indicators (KPIs) explained below, for growing natural search channel programs under management.

KPI #1 – Brand to Non-Brand Mix

What percentage of your natural search comes from brand keywords versus non-brand keywords? What does it mean if your keyword curve is dominated by brand terms? If you discover that most of your traffic is coming from searches for your brand, we view this as symptomatic of a larger problem lying just beneath the surface - that very few of your pages are actually yielding traffic.

Many retailers find 95% of their search traffic comes for brand terms with, not coincidentally, a very small percentage of their site powering that brand traffic. Whereas what you may find, when you have done some search optimization, is a very different distribution curve. You may find that 40% of your pages are yielding traffic, but that 60% of that traffic is unbranded keyword traffic. This in fact, is the core hypothesis of Page Yield Theory – unbranded keyword traffic volume grows as the number of pages yielding natural search traffic grows.

KPI #2 – Unique Pages

Just how big is your website? How many pages are there? This is a critical metric for establishing a yield management foundation. You may be tempted to approximate this number using a product count. This will most likely significantly understate your actual pages. Or you might want to use a search engine’s reported index. This too will return mixed results, as each engine indexes dramatically different amounts of pages for each website.

However, search engines are in the business of discovering pages. So we recommend using the number of pages that bots (such as googlebot) are able to crawl over a 30 – 60 day period as the most useful proxy for how many unique pages are available on your site. (This assumes your URLs are in a condition that permits bots to crawl deeply. If they are not, you may need to resort to an approximation.)

As a comparison, the average merchant in our research had roughly 73,000 uniquely crawled pages.

Note: Most analytical packages do not track page crawl metrics because they rely on Javascript to track user actions. Since search engine bots do not have Javascript enabled, their crawl activity goes unregistered – leaving marketers blind to the dynamics of their “search conversion funnel.

KPI #3 – Pages Yielding Traffic

What percentage of those pages yield search traffic? Is it 10%? 90%? This ratio essentially dictates the length (the X axis) of your unbranded keyword long tail, and suggests remaining potential.

With this KPI in hand, you understand your missed opportunity cost, and monitor your progress throughout your NSO engagement.

If you do not know your amount of unique pages or yielding pages, the above matrix can be consulted for an estimate. It was developed using data from the average merchant in our research and illustrates the inverse relationship between Page Yield and Brand / Non-Brand Traffic. Simply seek your current level of brand and non-brand traffic to estimate how many pages are accounting for that traffic.

The average merchant in our study had 14% of pages yielding traffic.

KPI #4 – Keywords per Page Yield

Now that you have an idea of your Page Yield rate, how many keywords does each of those producing pages yield over the course of a month – 2 keywords? 10? This is the fourth KPI: keyword yield. This KPI is responsible for creating scale. That is, the more keywords each yielding page attracts or targets, the longer your tail.

The average merchant in our study found 2.4 keywords produced per yielding page.

KPI #5 – Visitor per Keyword Yield

This fifth KPI – Visitor per Keyword Yield – is basically determined by how highly a page ranks for a keyword, and tells you how much traffic each keyword drives. This metric determine the height or thickness of your long tail.

The average merchant in our study experienced 1.9 visitors per keyword.

KPI #6 – Index-to-Crawl Ratios

The first step in the search conversion funnel is converting a crawled page to an indexed page. While we do not put significant stock in the reported engine indexes, this KPI can illustrate engine differences or potential trouble-spots.

For instance, the average merchant in our study saw a 3:1 index-to-crawl ratio in Google. That is, for every page that was crawled, there were 3 in the index, as strange as it sounds. If a merchant finds the index shrinking to say, only 50% of the crawled pages, that may be a signal of crawl pattern changes, or perhaps pages are being shifted into the supplemental index.

KPI #7 – Engine Yield

A final KPI to monitor is engine yield. Each engine has different audience sizes. How do you fairly compare the referral traffic you get from each? Simply calculate how much referral traffic each sends for every page it crawls or consumes. Compare engine by engine. What we have found is that MSN and Yahoo! tend to crawl a lot more pages, but the yield per crawled page from Google is typically significantly higher.

Reconstructing the Tail

Once known, these KPIs can be reassembled to create a vivid picture of your performance and empower smarter decisions. For instance, brand and non-brand traffic composition (KPI #1) provide a baseline on the effectiveness of your channel yield. This can be contrasted against your total available pages (KPI #2) and actual yielding pages (KPI #3) to understand the gap between current and potential performance. The rate at which these pages yield keywords (KPI #4), times the visitor driven by each of those keywords (KPI #5) form the length, and height, of your tail, and can each be optimized individually. Meanwhile your index-to-crawl ratio (KPI #6) and engine yield (KPI #7) help quantify your site’s effectiveness at converting web pages into units of “search produce.”

For instance, if the number of pages yielding traffic is low (under 10%), it may be time for some basic, scalable, URL rewriting to maximize the flow of link juice through your internal link text. On the other hand, if 80% of pages are yielding traffic at a rate of one keyword per page and one visitor per keyword, it may be time to conduct some auto-generated title tag tests to scalably lift rankings of thousands of pages, by incorporating more relevant keywords per page. And perhaps a pinch of organic link building via RSS may be in order to power-up the link juice that flows through the inbound link text. Each situation is unique and requires a custom approach to optimization.

The secret to capturing the long tail of natural search is this: Fully maximize the page yield of your website. That is, develop a metrics-driven optimization process that empowers each page to be viewed as the authority on the subject – be it for “furniture slipcovers,” “men’s reindeer sweater,” “18 volt Makita power drill” or other. This is not without challenge; empowering thousands of pages to yield natural search traffic may seem daunting, but it can be achieved by focusing on these new KPIs, managing your NSO channel effort against these yield metrics, and committing to an iterative NSO testing program.

Increasingly, the natural search game revolves around getting your brand associated with unbranded terms. While these new KPIs do not change the basics of NSO, they can provide the context and management discipline that enable marketers to make more informed decisions how to best spend time and budget optimizing your long tail.

For Immediate Release

Lost Shoppers Found by E-retailers Using Long Tail Dynamics

MADISON, WI (December 21, 2006) Lost shoppers are being found through unbranded keyword searches by the e-commerce clients of Netconcepts, a global Web technology company specializing in natural search optimization consulting and search-friendly E-Business design. Netconcept’s clients are increasing their e-retail traffic through natural search optimizations that don’t depend on “brand name” keywords.

Netconcepts released a white paper last month, “Chasing the Long Tail of Natural Search,” which details the dynamics of the long tail phenomena and how e-commerce marketers can capture unbranded keyword searches. The study reveals the compelling truth that the unbranded market opportunity is 40 times richer than branded searches.

“Chasing the Long Tail of Natural Search” provides a framework to help merchants answer questions such as: how to capture this market, what type of traffic they should anticipate, and how to estimate the “unbranded keyword” market potential. The white paper is available at no charge on the company’s website.

“Consumers searching the Internet for products frequently know what they want to buy, but don’t include any particular brand in their search query. They often enter basic keywords such as ‘athletic shoes’ or ‘gift baskets,’” said Brian Klais, Vice President, Search. “For the average merchant we studied, there are roughly 7,000,000 unbranded keyword searches conducted per month. To capture these searches, we work with our clients to optimize their websites using long tail dynamics and a new set of key performance indicators (KPI). Our white paper details the concept of the long tail and provides the who, what, where and why this is important to marketers.”

Griffin discusses why the (over discussed, under explained) 80/20 rule has such validity on search engine marketing of today’s business. He also summarizes Netconcepts’ key tactics to capturing the Long Tail of Natural Search.

Read this entire article and broaden your thinking of current Long Tail research.